Commonly submersed, vertically removable servers in rack

ABSTRACT

Apparatus, systems, and method for efficiently cooling computing devices having heat-generating electronic components, such as, for example, independently operable servers, immersed in a dielectric liquid coolant in a tank.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority pursuant to 35 U.S.C. 119 to thefollowing U.S. provisional patent applications:

Ser. No. 61/188,589 entitled LIQUID SUBMERGED, HORIZONTAL COMPUTERSERVER RACK filed Aug. 11, 2008;

Ser. No. 61/163,443 entitled LIQUID SUBMERGED, HORIZONTAL COMPUTERSERVER RACK filed Mar. 25, 2009; and

Ser. No. 61/165,470 entitled LIQUID SUBMERGED, HORIZONTAL COMPUTERSERVER RACK filed Mar. 31, 2009.

FIELD OF INVENTION

This application concerns cooling of heat-generating electronics suchas, for example, rack mounted servers in data centers.

BACKGROUND

In 2006, data centers in the United States (U.S.) accounted for about1.5% (about $4.5 billion) of the total electricity consumed in the U.S.This data center electricity consumption is expected to double by 2011.More than one-third of data center electricity consumption is forcooling servers, which could equate to more than about 1% of all U.S.electricity consumed by 2011. Electricity, personnel, and constructioncosts continue to increase and server hardware costs are decreasing,making the overall cost of cooling a large and growing part of the totalcost of operating a data center.

The term “data center” (also sometime referred to as a “server farm”)loosely refers to a physical location housing one or “servers.” In someinstances, a data center can simply comprise an unobtrusive corner in asmall office. In other instances, a data center can comprise severallarge, warehouse-sized buildings enclosing tens of thousands of squarefeet and housing thousands of servers. The term “server” generallyrefers to a computing device connected to a computing network andrunning software configured to receive requests (e.g., a request toaccess or to store a file, a request to provide computing resources, arequest to connect to another client) from client computing devices,includes PDAs and cellular phones, also connected to the computingnetwork. Such servers may also include specialized computing devicescalled network routers, data acquisition equipment, movable disc drivearrays, and other devices commonly associated with data centers.

Typical commercially-available servers have been designed for aircooling. Such servers usually comprise one or more printed circuitboards having a plurality of electrically coupled devices mountedthereto. These printed circuit boards are commonly housed in anenclosure having vents that allow external air to flow into theenclosure, as well as out of the enclosure after being routed throughthe enclosure for cooling purposes. In many instances, one or more fansare located within the enclosure to facilitate this airflow.

“Racks” have been used to organize several servers. For example, severalservers can be mounted within a rack, and the rack can be placed withina data center. Any of various computing devices, such as, for example,network routers, hard-drive arrays, data acquisition equipment and powersupplies, are commonly mounted within a rack.

Data centers housing such servers and racks of servers typicallydistribute air among the servers using a centralized fan (or blower). Asmore fully described below, air within the data center usually passesthrough a heat exchanger for cooling the air (e.g., an evaporator of avapor-compression cycle refrigeration cooling system (or “vapor-cycle”refrigeration), or a chilled water coil) before entering a server. Insome data centers, the heat exchanger has been mounted to the rack toprovide “rack-level” cooling of air before the air enters a server. Inother data centers, the air is cooled before entering the data center.

In general, electronic components of higher performing servers dissipatecorrespondingly more power. However, power dissipation for each of thevarious hardware components (e.g., chips, hard drives, cards) within aserver can be constrained by the power being dissipated by adjacentheating generating components, the airflow speed and airflow paththrough the server and the packaging of each respective component, aswell as a maximum allowable operating temperature of the respectivecomponent and a temperature of the cooling air entering the server asfrom a data center housing the server. The temperature of an air streamentering the server from the data center, in turn, can be influenced bythe power dissipation and proximity of adjacent servers, the airflowspeed and the airflow path through a region surrounding the server, aswell as the temperature of the air entering the data center (or,conversely, the rate at which heat is being extracted from the airwithin the data center).

In general, a lower air temperature in a data center allows each servercomponent to dissipate a higher power, and thus allows each server todissipate more power and operate at a level of hardware performance.Consequently, data centers have traditionally used sophisticated airconditioning systems (e.g., chillers, vapor-cycle refrigeration) to coolthe air (e.g., to about 65° F.) within the data center for achieving adesired performance level. By some estimates, as much as one watt can beconsumed to remove one watt of heat dissipated by an electroniccomponent. Consequently, as energy costs and power dissipation continueto increase, the total cost of cooling a data center has also increased.

In general, spacing heat-dissipating components from each other (e.g.,reducing heat density) makes cooling such components less difficult (andless costly when considering, for example, the cost of cooling anindividual component in a given environment) than placing the samecomponents placed in close relation to each other (e.g., increasing heatdensity). Consequently, data centers have also compensated for increasedpower dissipation (corresponding to increased server performance) byincreasing the spacing between adjacent servers.

In addition, large-scale data centers have provided several coolingstages for cooling heat dissipating components. For example, a stream ofcoolant, e.g., water, can pass over an evaporator of a vapor-compressionrefrigeration cycle cooling system and be cooled to, for example, about44° F. before being distributed through a data center for cooling airwithin the data center.

The power consumed by a chiller can be estimated using information fromstandards (e.g., ARI 550/590-98). For example, ARI550/590-98 specifiesthat a new centrifugal compressor, an efficient and common compressorused in high-capacity chillers, has a seasonal averageCoefficient-of-Performance (“COP”) from 5.00 to 6.10, depending on thecooling capacity of the chiller. This COP does not include powerconsumed by an evaporative cooling tower, which can be used for coolinga condenser in the refrigeration cycle cooling system and generally hasa COP of 70, or better. The combined COP for a typical system isestimated to be about 4.7.

According to some estimates, some state-of-the-art data centers arecapable of cooling only about 150 Watts-per-square-foot, as opposed tocooling the more than about 1,200 Watts-per-square-foot that couldresult from arranging servers to more fully utilize available volume(e.g., closely spacing servers and racks to more fully utilizefloor-to-ceiling height and floor space) within existing data centers.Such a low cooling capacity can significantly add to the cost ofbuilding a data center, since data centers can cost as much as about$250 per-square-foot to construct.

As the air-cooling example implies, commercially available methods ofcooling have not kept pace with increasing server and data-centerperformance needs, or the corresponding growth in heat density. As aconsequence, adding new servers to existing data centers has becomedifficult and complex given the effort expended to facilitate additionalpower dissipation, such as by increasing an existing data center's airconditioning capacity.

Various alternative approaches for cooling data centers and theirservers, e.g., using liquid cooling systems, have met with limitedsuccess. For example, attempts to displace heat from a microprocessor(or other heat-generating semiconductor-fabricated electronic devicecomponent, collectively referred to herein as a “chip”) for remotelycooling the chip have been expensive and cumbersome. In these systems, aheat exchanger or other cooling device, has been placed in physicalcontact (or close physical relation using a thermal-interface material)with the package containing the chip. These liquid-cooled heatexchangers have typically defined internal flow channels for circulatinga liquid internally of a heat exchanger body. However, componentlocations within servers can vary from server to server. Accordingly,these liquid-cooling systems have been designed for particular componentlayouts and have been unable to achieve large-enough economies of scaleto become commercially viable.

Research indicates that with state-of-the-art cooling, PUEs (as definedon page 10 hereinafter) of 1.4 might be attainable by 2011. However thecosts to capitalize such cooling were not mentioned, and indicatorssuggest that saving electricity requires expensive equipment.

Immersion cooling of electronic components has been attempted inhigh-performance (e.g., computer gaming) applications, but has notenjoyed widespread commercial success. Previous attempts at immersioncooling has submerged some, and in some instances all, componentsmounted to a printed circuit board in a dielectric fluid using ahermetically sealed enclosure to contain the fluid. Such systems havebeen expensive, and offered by a limited number of suppliers. Largescale data centers generally prefer to use “commoditized” servers andtend to not rely on technologies with a limited number of suppliers.

Control systems have been used to increase cooling rates for a pluralityof computers in response to increased computational demand. Even so,such control systems have controlled cooling systems that dissipate heatinto the data center building interior air (which in turns needs to becooled by air conditioning), or directly use refrigeration as a primarymode of heat dissipation. Refrigeration as a primary mode of cooling,directly or indirectly, requires significant amounts of energy.

Two-phase cooling systems have been attempted, but due to technicalcomplexity, they have not resulted in cost-effective products orsufficiently low operating costs to justify investing intwo-phase-cooling capital. Still other single- and two-phase coolingsystems bring the coolant medium to an exterior of the computer, butreject heat to a cooling medium (e.g., air) external to the computer andwithin the data center (e.g., within a server room). Accordingly, eachmethod of server or computer cooling currently employed or previouslyattempted have been prohibitively expensive and/or insufficient to meetincreasing cooling demands of computing devices.

Indirectly, many researchers have tried to reduce the power ofindividual components such as the power supply and CPU. Although chipscapable of delivering desirable performance levels while operating at alower relative power have been offered by chip manufacturers, such chipshave, to date, been expensive. Consequently, cooling approaches to datehave resulted in one or more of a high level of electricity consumption,a large capital investment and an increase in hardware expense.

Therefore, there exists the need for an effective, efficient andlow-cost cooling alternative for cooling electronic components, such as,for example, rack-mounted servers.

SUMMARY OF INVENTION

Briefly, the present invention provides novel apparatus, systems, andmethods for efficiently cooling computing devices having heat-generatingelectronic components, such as, for example, independently operableservers immersed in a dielectric liquid coolant in a tank.

The system may include at least one tank defining an interior volume andhaving a coolant inlet for receiving a dielectric liquid coolant withinthe interior volume and having a coolant outlet for allowing thedielectric liquid coolant to flow from the interior volume, the coolantinlet and the coolant outlet being fluidly coupled to each other; one ormore mounting members positioned within the interior volume andconfigured to mountably receive a plurality of independently operableservers; a dielectric liquid coolant; a heat exchanger fluidly coupledto the coolant outlet of the at least one tank, the heat exchanger beingdistally located from the tank; a pump fluidly coupled to the heatexchanger and the interior volume of the at least one tank, the pumpbeing configured for pumping the liquid coolant through a fluid circuitcomprising a first circuit portion extending from the coolant inlet ofthe tank to each server, a second circuit portion extending from eachrespective server to the coolant outlet, a third circuit portionextending from the coolant outlet to the heat exchanger, and a fourthportion extending from the heat exchanger to the coolant inlet; acontroller for monitoring the temperature of the dielectric liquidcoolant at at least one location within the fluid circuit and foradjusting the flow of the dielectric liquid coolant through the fluidcircuit in order that the dielectric liquid coolant is maintained at anelevated temperature as it exits the second circuit portion of the fluidcircuit; wherein the at least one tank is configured for containing thedielectric liquid coolant within the interior volume such that, when theplurality of servers are mountably received therein, each server issubmerged within the dielectric liquid coolant for sufficiently coolingeach respective server while maintaining the exiting heated liquidcoolant at the elevated temperature to reduce the amount of energyconsumed to sufficiently cool each of the plurality of servers.

Alternatively, the cooling system includes at least one tank defining anopen interior volume; one or more mounting members positioned within theopen interior volume and configured to mountably receive a plurality ofindependently operable servers within the interior volume; a dielectricliquid coolant circulating in a first fluid circuit through theplurality of servers; a secondary cooling system having a cooling fluidflowing in a second fluid circuit wherein the secondary cooling systemrejects heat from the cooling fluid; a coupler located within the atleast one tank for thermally coupling heated dielectric coolant from theportion of the first fluid circuit exiting the plurality of serverswithin the tank to the cooling fluid in the second fluid circuit forrejecting heat from such heated dielectric coolant; a controller formonitoring the temperature of the dielectric liquid coolant at at leastone location within the first fluid circuit and for adjusting the flowof the cooling fluid through the second fluid circuit in order that theheated dielectric liquid coolant exiting the plurality of servers ismaintained approximately at an elevated temperature wherein the elevatedtemperature is a temperature significantly higher than the typicalcomfortable room temperature for humans and lower than the maximumpermissible temperature of the most sensitive heat generating electroniccomponent in the plurality of servers; wherein the at least one tank isconfigured for containing the dielectric liquid coolant within theinterior volume such that, when the plurality of servers are mountablyreceived therein, at least a substantial portion of each server issubmerged within the dielectric liquid coolant for sufficiently coolingeach respective server when the tank is sufficiently full of the liquidcoolant maintaining the liquid coolant [exiting] the plurality ofservers at approximately the elevated temperature to reduce the amountof energy consumed to sufficiently cool each respective server.

Alternatively, the cooling system may include at least one tank definingan open interior volume; one or more mounting members positioned withinthe open interior volume and configured to mountably receive a pluralityof independently operable servers within the interior volume; adielectric liquid coolant circulating in a first fluid circuit throughthe plurality of servers; a secondary cooling system having a coolingfluid flowing in a second fluid circuit wherein the secondary coolingsystem rejects some of the heat from the cooling fluid; a couplerlocated within the at least one tank for thermally coupling heateddielectric coolant from the portion of the first fluid circuit exitingthe plurality of servers within the tank to the cooling fluid in thesecond fluid circuit for rejecting some of the heat from such heateddielectric coolant; a controller for monitoring the temperature of thedielectric liquid coolant at at least one location within the firstfluid circuit and for adjusting the flow of the cooling fluid throughthe second fluid circuit in order that the heated dielectric liquidcoolant exiting the plurality of servers is maintained approximately atan elevated temperature wherein the elevated temperature is atemperature significantly higher than the typical comfortable roomtemperature for humans and lower than the maximum permissibletemperature of the most sensitive heat generating electronic componentin the plurality of servers; wherein the at least one tank is configuredfor containing the dielectric liquid coolant within the interior volumesuch that, when the plurality of servers are mountably received therein,each server is submerged within the dielectric liquid coolant forsufficiently cooling each respective server and maintaining the liquidcoolant exiting the plurality of servers at approximately the elevatedtemperature to reduce the amount of energy consumed to sufficiently cooleach respective server.

The fixture or server rack apparatus includes at least one tank definingan open interior volume and having a coolant inlet for receiving adielectric liquid coolant within the open interior volume and having acoolant outlet for allowing the coolant to flow from the open interiorvolume, the coolant inlet and the coolant outlet being fluidly coupledto each other; and one or more mounting members positioned within theinterior volume and configured to mountably receive a plurality ofservers in a vertical orientation within the interior volume forminimizing the footprint of the server relative to the ground and withthe front of the server facing upward for easy installation and removalof each of the plurality of servers without removing or disturbing anyother server; wherein the at least one tank is configured for containinga dielectric liquid coolant within the interior volume such that, when aplurality of servers are mountably received therein, each server beingmountably received is submerged within the dielectric liquid coolant forsufficiently cooling each respective server when the tank issufficiently full of the liquid coolant.

A server room fluidly connected to a first heat exchanger distallylocated from the server room contains the apparatus described above,including at least one tank defining an interior volume for containing adielectric liquid coolant and one or more mounting members positionedwithin the interior volume and configured to mountably receive aplurality of independently operable servers. The server room alsocontains a plurality of independently operable servers wherein each ofthe plurality of servers is mountably received by the one or moremounting members such that each of the respective servers is submergedin a volume of dielectric liquid coolant for absorbing heat from eachrespective one of the plurality of servers. The server room furthercontains at least one coupler for thermally coupling the heateddielectric liquid coolant heated to the heat exchanger for rejecting atleast some of the heat absorbed by the dielectric liquid coolant fromeach of the plurality of servers. The heat exchanger may be associatedwith a secondary cooling system. The coupler may include a fluid couplerfor fluidly coupling the dielectric liquid coolant to the first heatexchanger. Alternatively, the coupler includes a heat exchanger locatedinternal to the tank and thermally coupled to the dielectric liquidcoolant heated by the servers and a secondary fluid circuit with asecond cooling fluid in fluid connection between the distally locatedheat exchanger and the internally located heat exchanger wherein thedielectric liquid coolant differs from the cooling fluid wherein thedistally located heat exchanger is thermally coupled to the coolingfluid flowing in the secondary fluid circuit such that the distallylocated heat exchanger rejects heat from the cooling fluid which thecooling fluid has absorbed from the heated dielectric liquid coolant atthe coupler.

A method of cooling a plurality of independently operable serversincludes flowing a dielectric liquid coolant in a fluid circuit throughthe plurality of servers immersed within the dielectric liquid coolantfor absorbing at least a portion of any heat being dissipated by each ofthe respective servers; monitoring the temperature of the liquid coolantat at least one location within the fluid circuit; determining theoptimum elevated temperature of the heated dielectric liquid coolant asit exits the plurality of servers such that the liquid coolantsufficiently cools the plurality of servers while reducing the amount ofenergy consumed to sufficiently cool each respective server, wherein theelevated temperature is a temperature significantly higher than thetypical comfortable room temperature for humans and lower than themaximum permissible temperature of the most sensitive heat generatingelectronic component in the plurality of servers; periodicallydetermining by a controller the amount of energy needed to reject theabsorbed heat for cooling the plurality of servers; thermally couplingthe dielectric liquid coolant heated by the plurality of servers to aheat exchanger distally located from the tank; and rejecting at least aportion of the heat absorbed by the liquid coolant. In response to theperiodic determination of the amount of energy needed to reject the heatabsorbed by the dielectric liquid coolant from the servers by acontroller, the method may also include the step of periodicallyadjusting the amount of heat rejected through the heat exchanger suchthat the dielectric liquid coolant exiting the plurality of servers atthe elevated temperature sufficiently cools the plurality of serverswhile reducing the amount of energy consumed to sufficiently cool eachrespective server.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention(s), and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawings, in which:

FIG. 1A illustrates one embodiment of an exemplary system forefficiently cooling a plurality of independently operable servers;

FIG. 1B illustrates an alternative embodiment of an exemplary system forefficiently cooling a plurality of independently operable servers;

FIG. 2 illustrates the system of FIG. IA in more detail;

FIG. 3 illustrates a perspective view of an exemplary immersion-cooledrack having a plurality of independently operable servers mountedtherein.

FIG. 4 illustrates a top plan view of the immersion-cooled rack shown inFIG. 3.

FIG. 5 illustrates an end elevation view of the immersion-cooled rackshown in FIG. 3.

FIG. 6 illustrates a side elevation view of the immersion-cooled rackshown in FIG. 3.

FIG. 7 illustrates an end elevation view of an alternativeimmersion-cooled rack having a plurality of independently operableservers installed therein.

FIG. 8 illustrates a top plan view of the immersion-cooled rack shown inFIG. 7.

FIG. 9 illustrates an end elevation view of another alternativeimmersion-cooled rack having a plurality of independently operableservers mounted therein.

FIG. 10 illustrates an end elevation view of yet another alternativeimmersion-cooled rack having a plurality of independently operableservers mounted therein.

FIG. 11 illustrates a perspective view of side-by-side immersion-cooledracks having a plurality of independently operable servers mountedtherein with the electrical connections to the servers shown.

FIG. 12A is perspective view of one version of a conventionalrack-mountable server that may be installed in the exemplaryimmersion-cooled server racks depicted in FIGS. 3 through 11;

FIG. 12B is an illustration of a hard drive of the conventionalrack-mountable server of FIG. 12A with a liquid-proof enclosure to beinserted around it;

FIG. 13 is an end elevation view of the immersion-cooled server racks ofFIG. 11;

FIG. 14 is another end elevation view of the immersion-cooled serverracks of FIG. 11 showing the flow of the liquid coolant;

FIG. 15 is a schematic illustration of a system for cooling a pluralityof immersion-cooled server racks of the type shown in, for example, FIG.3 and installed in a server room.

FIG. 16 illustrates an exemplary method of cooling one or moreindependently operable servers immersed in a tank of liquid coolantemploying the systems of FIG. 1A or 1B;

FIG. 17A illustrates the physical steps in the method of cooling one ormore independently operable servers immersed in a tank of liquid coolantemploying the system of FIG. 1A; and

FIG. 17B illustrates the computer controller-based steps in the methodof cooling one or more independently operable servers immersed in a tankof liquid coolant employing the system of FIG. 1A.

DETAILED DESCRIPTION

The following describes apparatus, systems, and methods for efficientlycooling computing devices having heat-generating electronic components,such as, for example, independently operable servers at least partiallyimmersed in a dielectric liquid coolant in a tank. The principles of theinvention(s) embodied therein and their advantages are best understoodby referring to FIGS. 1-17.

As used herein, the term “server” generally refers to a computing deviceconnected to a computing network and running software configured toreceive requests (e.g., a request to access or to store a file, arequest to provide computing resources, a request to connect to anotherclient) from client computing devices, includes PDAs and cellularphones, also connected to the computing network. Such servers may alsoinclude specialized computing devices called blade servers, networkrouters, data acquisition equipment, movable disc drive arrays, andother devices commonly associated with data centers.

As used herein, “independently operable” means capable of usefullyfunctioning without regard to an operational status of an adjacentcomponent. As used herein, an “independently operable server” means aserver that is capable of usefully functioning (e.g., powered orunpowered, connected to a network or disconnected from the network,installed in a rack or removed from a rack, and generally used for thepurposes for which servers are generally used) without regard to anoperational status of an adjacent server (e.g., powered or unpowered,connected to the network or disconnected from the network, installed inthe rack or removed from the rack, and whether usable for the purposesfor which servers are generally used). Operation of independentlyoperable servers can be influenced (e.g., heated) by one or moreadjacent servers, but as used herein, an independently operable servergenerally functions regardless of whether an adjacent server operates oris operable.

As used herein, the term “liquid coolant” may be any sufficientlynon-conductive liquid such that electrical components (e.g., amotherboard, a memory board, and other electrical and/or electroniccomponents designed for use in air) continue to reliably function whilesubmerged without significant modification. A suitable liquid coolant isa dielectric liquid coolant, including without limitation vegetable oil,mineral oil (otherwise known as transformer oil), or any liquid coolanthave similar features (e.g., a non-flammable, non-toxic liquid withdielectric strength better than or nearly as comparable as air.

As used herein, “fluid” means either a liquid or a gas, and “coolingfluid” means a gas or liquid coolant typically used for heat-rejectionor cooling purposes. As used herein, a liquid coolant is a subset of theuniverse of cooling fluids, but a cooling fluid may be a dielectric ornon-dielectric liquid or gas, such as, for example, a conventional airconditioning refrigerant.

PUE means “power usage effectiveness”, which is a ratio of the totalpower used by a data center divided by the power used by the server, andis a measure of energy efficiency.

COP means the “coefficient of performance”, a ratio of heat removed towork used. For instance, a COP of 10 would mean that 10 Watts of heatare removed using 1 Watt of work.

VCC means “vapor compression cycle”, the thermal process most commonlyused for air conditioning.

Poor overall efficiency of heretofore commercially available coolingtechnologies contributes to overall costs of cooling servers used bydata centers. As disclosed herein, applicants have discovered that theirreversibility's contributing to this poor overall efficiency can bereduced, reducing the overall cost of cooling servers (as well as thecorresponding cost of operating data centers).

As between two bodies (or fluids) at different temperatures, heat flowsfrom the higher-temperature body to the lower-temperature body. For agiven amount of transferred heat, such heat transfer is lessirreversible (e.g., the associated energy retains more “usefulness,” oris of a “higher quality”) when both temperatures are higher as comparedto a heat transfer process occurring at lower temperatures. Methods,systems, and apparatus are disclosed for efficiently coolingheat-generating electronic components, as by transferring heat from thecomponents at a first temperature (e.g., about 158° F. in someinstances) to a liquid coolant at a “high” temperature (e.g., adielectric liquid coolant such as, for example, mineral oil at atemperature of, for example, about 105° F.). Such heat transfer from theheat-generating components at the first temperature to a coolant at a“high” temperature can be less irreversible than transferring the samequantity of heat from the components at the first temperature to acoolant at a “low” temperature (e.g., air at a temperature of, forexample, 65° F.).

The methods, systems, and apparatus disclosed herein take advantage ofthis thermodynamic principle to improve the overall efficiency ofcooling electronic components, as can be applied to, for example,independently operable servers of the type commonly used in a datacenter. Such improved cooling efficiency can reduce the overall cost ofoperating a data center by reducing electricity consumed for coolingpurposes.

In some disclosed embodiments, the reduced temperature differences(resulting in lower irreversibility) allows for heat to be recaptured.In other embodiments, the reduced temperature differences reduces (oraltogether removes) the need for refrigeration. In all of the disclosedembodiments, the corresponding cooling cycle efficiency of the coolingsystem increases as compared to conventional, commercially availablecooling cycles.

Overview

FIG. 1A and FIG. 1B depict alternative exemplary systems 100 and 200,respectively, for cooling one or more independently operable serverscontaining heat-generating electronic components, such as can bearranged in one or more server racks, for example, in a data center.Some disclosed systems and methods reduce the temperature differencebetween heat generating (or dissipating) components and a cooling medium(also referred to herein as “coolant” or “liquid coolant”) used to coolthe components by maintaining a coolant temperature (e.g., an averagebulk fluid temperature) at an acceptably elevated temperature comparedto conventional cooling technologies. Such an elevated coolanttemperature can reduce the power consumed for cooling purposes (e.g.,heat can be more readily rejected from a “high-temperature” coolant tothe environment than from a “low-temperature” coolant).

FIG. 1A illustrates one embodiment of cooling system 100 for cooling arack of independently operable servers. The system 100 includes a tub ortank 110 containing a dielectric liquid coolant into which a pluralityof servers 120 may be immersed. Mounting members or rails to bedescribed hereinafter are positioned within the interior volume of thetank 110 and are configured to receive and mount the plurality ofservers 120 as a rack of servers into the tank 110. Such a tank 110 mayhave an opening for access to each of the servers mounted in the rack.At least a portion of each server 120 is submerged within the dielectricliquid coolant for sufficiently cooling each respective server when thetank 110 is sufficiently full of the liquid coolant. Preferably, each ofthe servers during operation is completely submerged within thedielectric liquid coolant.

The liquid coolant heated by the servers 120 in the server rack is thenfluidly coupled through suitable piping or lines to a pump 130, whichpumps the heated liquid coolant through suitable piping or lines to aremotely or distally located heat exchanger 140 associated with aheat-rejection or cooling apparatus 150. The distally heat exchanger 140rejects the heat from the incoming heated liquid coolant and fluidlycouples the cooled liquid coolant through a return fluid line or piping170 back into the tank 110. Thus, at least a portion of the liquidcoolant completes a fluid circuit through the servers 120 in the tank110, pump 130, heat exchanger 140, and back into the tank 110. The heatrejected from the heated liquid coolant through the heat exchanger 140may then be selectively used by alternative heat rejection or coolingapparatus 150 to be described hereinafter to dissipate, recover, orbeneficially use the rejected heat depending on the differentenvironmental conditions and/or server operating conditions to which thesystem is subject.

The system 100 includes a computer controller 180 of conventional designwith suitable novel applications software for implementing the methodsof the present invention. The controller 180 may receive monitor signalsof various operational parameters from various components of the coolingsystem 100 and the environment and may generate control signals tocontrol various components of the cooling system to maintain the heatedliquid coolant exiting the servers in the tank at a specific elevatedtemperature in order to sufficiently cool each of the servers whilereducing the total amount of energy needed to cool the servers.Particularly, the controller 180 monitors the temperature of the liquidcoolant at at least one location within the fluid circuit, for examplewhere the heated liquid circuit exits the plurality of servers. Thecontroller 180 may also monitor the temperature of the heat-generatingelectronic components in the servers in the server racks by electricallyconnecting the controller 180 to the diagnostic output signals generatedby conventional rack-mountable servers. The controller may also monitorthe flow of the dielectric liquid coolant. Based upon such information,the controller 180 may output signals to the pump 130 and heat rejectionor cooling apparatus 150 to adjust the flow of the liquid coolantthrough the fluid circuit and the amount of the heat being rejected bythe heat rejection or cooling apparatus 150 for sufficiently coolingeach respective server while maintaining the heated liquid coolantexiting the servers at the elevated temperature to reduce the amount ofenergy consumed to sufficiently cool each of the servers in the serverrack.

FIG. 1B illustrates one embodiment of an alternative cooling system 200for cooling a rack of independently operable servers. The system 200includes a tub or tank 210 containing a liquid dielectric coolant intowhich a plurality of servers 120 (not shown) can be immersed. Mountingmembers to be described hereinafter are positioned within the interiorvolume of the tank 210 and are configured to receive and mount theplurality of servers 120 as a rack of servers into the tank 210. Such atank 210 may have an open top for access to each of the servers mountedin the rack. At least a portion of each server 120 is submerged withinthe dielectric liquid coolant for sufficiently cooling each respectiveserver when the tank 210 is sufficiently full of the liquid coolant.Preferably, each of the servers during operation is completely submergedwithin the dielectric liquid coolant.

Unlike the cooling system 100, heated dielectric liquid coolant does notflow outside the tank 210. Instead, the fluid circuit 270 of the flowingdielectric liquid coolant is completely internal to the tank 210. Athermal coupling device 280, such as a heat exchanger, is mounted withinthe tank 210 within the fluid circuit through the servers so that atleast a portion of the heated dielectric liquid coolant flow exiting theservers flows through the thermal coupling device 280. Cooled dielectricliquid coolant exits the coupling device 280 and at least a portion ofthe cooled dielectric coolant circulates in the internal fluid circuit270 back through the servers.

The system 200 includes a secondary heat rejection or cooling apparatus250 having a cooling fluid, such as a gas or liquid flowing in piping orlines, forming a second fluid circuit 290 wherein the secondary coolingapparatus 250 includes an associated remotely or distally located heatexchanger (not shown) that rejects heat from the cooling fluid in thesecond fluid circuit through the distally remote heat exchanger.

The heat rejected from the heated cooling fluid in the second fluidcircuit through the heat exchanger associated with the secondary coolingapparatus 250 may then be selectively dissipated, recovered, orbeneficially used depending on the different environmental conditionsand/or server operating conditions to which the system is subject.

The system 200 includes a computer controller 280 with suitable novelapplications software for implementing the methods of the presentinvention. The controller 180 may receive monitor signals of variousoperational parameters from various components of the cooling system 200and the environment and may generate control signals to control variouscomponents of the cooling system to maintain the heated liquid coolantexiting the servers in the tank 210 at a specific elevated temperaturein order to sufficiently cool each of the plurality of servers whilereducing the total amount of energy needed to cool the servers.Particularly, the controller 280 monitors the temperature of the liquidcoolant at at least one location within the internal fluid circuit, forexample, where the heated liquid circuit exits the servers immersed inthe tank. The controller 280 may also monitor the temperature of theheat-generating electronic components in the servers in the server racksby electrically connecting the controller to the diagnostic outputsignals generated by conventional rack-mountable servers. The controllermay also monitor the flow and temperature of the cooling fluid in theexternal fluid circuit 290. Based upon such information, the controller180 may output signals to the heat rejection or cooling apparatus 250 toadjust the flow of the cooling liquid through the external fluid circuitand the amount of the heat being rejected by the heat rejection orcooling apparatus 250 for sufficiently cooling each respective serverwhile maintaining the heated liquid coolant exiting the servers at theelevated temperature to reduce the amount of energy consumed tosufficiently cool each of the servers. Preferably, the elevatedtemperature is a temperature significantly higher than the typicalcomfortable room temperature for humans and lower than the maximumpermissible temperature of the most sensitive heat generating electroniccomponent in the servers.

As previously described, a computer controller is used control differentcomponents of the cooling system to maintain the exiting dielectricliquid coolant temperature at an acceptable elevated temperature. Bymaintaining the existing coolant at an elevated level, the coolingsystem may be used with a number of different techniques for using ordissipating the heat (e.g., heat recapture, low power heat dissipation,or refrigeration).

In some embodiments, an average bulk fluid temperature of the coolantcan be maintained at a temperature of about, for example, 105° F., whichis significantly higher than a typical room temperature, as well as themaximum average outdoor temperature by month in the U.S. (e.g., about75° F. during summer months). At a temperature of about 105° F., heatcan be rejected to the environment (e.g., the atmosphere or nearbycooling sources such as rivers) with little power consumed, orrecaptured as by, for example, heating the same or an adjacentbuilding's hot-water supply or providing indoor heating in coldclimates.

By maintaining a coolant temperature in excess of naturally occurringtemperatures, irreversibility's and/or temperature differences presentin a server cooling system may be reduced. A reduction inirreversibility's in a thermodynamic cycle tends to increase the cycle'sefficiency, and may reduce the overall power consumed for cooling theservers.

In a conventional cooling system, about one-half watt is consumed by thecooling system for each watt of heat generated in a component. Forexample, a cooling medium (e.g., air) can be cooled to about 65° F. andthe components to be cooled can operate at a temperature of about, forexample, 158° F. This large difference in temperature results incorrespondingly large inefficiencies and power consumption. In addition,the “quality” of the rejected heat is low, making the heat absorbed bythe cooling medium difficult to recapture after being dissipated by thecomponent(s). However, with a cooling medium such as air, such a largetemperature difference may be necessary in conventional systems in orderto achieve desired rates of heat transfer.

-   -   For example, one-dimensional heat transfer, Q_(1-D), can be        modeled as the quotient of a temperature difference, Δ T,        divided by a thermal resistance, R_(th)

$\left( {{i.e.},{Q_{1 - d} = \frac{\Delta \; T}{R_{th}}}} \right).$

Accordingly, for a given heat dissipation from a component, atemperature difference between the component and a stream of liquidcoolant needs to be larger for higher thermal resistance than for alower thermal resistance. Typically, a flow of gas (e.g., air) has ahigher thermal resistance value than a flow of liquid (e.g., adielectric liquid coolant). Accordingly, a gas cooling fluid typicallyrequires a larger temperature difference than a liquid coolant.

Illustrative Embodiments of the System and Apparatus

In FIG. 2, the cooling system 300 illustrates one embodiment of thecooling system 100 of FIG. 1A in more detail. The system 300 includes atub or tank 310 containing a liquid coolant into which a plurality ofservers 120 can be immersed. Mounting members to be describedhereinafter are positioned within the interior volume of the tank 310and are configured to receive and mount the plurality of servers as arack of servers into the tank 310. Such a tank 310 may have an openingfor access to each of the servers mounted in the rack. At least aportion of each server 120 is submerged within the liquid coolant forsufficiently cooling each respective server when the tank 310 issufficiently full of the liquid coolant. Preferably, each of the serversduring operation is completely submerged within the liquid coolant.

The liquid coolant heated by the servers 120 in the server rack is thenfluidly coupled through suitable piping or lines to a pump 330, whichpumps the heated liquid coolant through suitable piping or lines througha filter 360 to one or more fluid valves 390. The fluid valve 390 may beremotely controlled to connect the heated liquid coolant being pumpedthrough the collection piping from the tank 310 to a controller-selectedone of alternative remotely or distally located heat exchangersassociated with alternative heat rejection or cooling apparatus 350,such as an outside air radiator 352 permitting cooling with outsideambient atmospheric air, a refrigeration system 354, a heat recoverysystem 356, or an evaporative cooler 358. The distally located heatexchanger associated with a selected one of the alternative heatrejection or cooling apparatus 350 then rejects the heat from theincoming heated liquid coolant and fluidly couples the cooled liquidcoolant through a return fluid line or piping 370 back into the tank310. Thus, at least a portion of the liquid coolant completes a fluidcircuit through the servers 120 in the tank 310, pump 330, a heatexchanger associated with a heat-rejection apparatus 350, and backthrough piping 370 into the tank 310. The heat rejected from the heatedliquid coolant through the heat exchanger may then be used by theselected one of alternative heat rejection or cooling apparatus 350 todissipate, recover, or beneficially use the rejected heat depending onthe different environmental conditions and/or server operatingconditions to which the cooling system 300 is subject.

The cooling system 300 includes a computer controller 380 with suitableapplications software which may receive monitor signals of variousoperational parameters from various components of the system 300 and theenvironment and may generate control signals to control variouscomponents of the system 300 to maintain the heated liquid coolantexiting the servers in the tank at a specific elevated temperature inorder to sufficiently cool each of the plurality of servers whilereducing the total amount of energy needed to cool the servers. Similarto previous embodiments, the controller 380 monitors the temperature ofthe liquid coolant at at least one location within the fluid circuit,for example where the heated liquid coolant exits the plurality ofservers. The controller may also monitor the temperature of theheat-generating electronic components in the servers 120 in the serverracks by electrically connecting the controller 380 to the diagnosticoutput signals generated by conventional servers. The controller 380 mayalso monitor the flow of the liquid coolant through the tank and/orfluid circuit. Based upon such information, the computer controller mayoutput signals to the pump 330 and heat rejection or cooling apparatus350 to adjust the flow of the liquid coolant through the fluid circuitand the amount of the heat being rejected by the heat rejection orcooling apparatus 350 for sufficiently cooling each respective serverwhen the tank 310 is sufficiently full of the liquid coolant whilemaintaining the heated liquid coolant exiting the servers at theelevated temperature to reduce the amount of energy consumed tosufficiently cool each of the plurality of servers. In addition, thecontroller 380 also may operate an optimization program within theapplications software as discussed hereinafter to determine which of thealternative heat rejection apparatus 350 connected to the fluid valve390 provides the most efficient means of rejecting the heat from theheated liquid coolant given the environmental and server operatingconditions. It should be noted, however, that the cooling system 300does not necessarily require different methods of heat dissipation. Insome instances it may be more cost effective to only have one.

In FIGS. 3 thru 6, a suitable fixture or rack apparatus 400 forimmersing a rack of independently operable servers in a liquid coolant422 is depicted. The apparatus 400 includes a tub or tank 410 andmounting members for mounting the servers, as will be described in moredetail hereinafter. The tank 410 may be fabricated of steel, asufficiently strong plastic that is compatible with the dielectricliquid coolant used as a cooling medium, or other suitable material. Thetank 410 may face upward with an open top 430 to form an open interiorvolume and may be shaped to have a length L, width W, and height H withthe minimum footprint to insert multiple servers 120. Suitable mountingmembers may be used to mount the servers in the tank to form the serverrack 470 within the tank. The tank 410 may be shaped and the L, W, and Hdimensions sized such that multiple standard-sized servers, typicallymeasured in units of “U” or 1.75 inches (as shown in FIG. 4), can besupported without significant modification.

The tank is fabricated to have an inlet pipe or line 440 from a pipingsystem connected to a heat exchanger for the flow of lower temperatureor cooled liquid coolant into the tank 410 and an outlet pipe or line450 connected to collection piping for the flowing or pumping of heatedcoolant out of the tank to the external heat exchanger associated withone or more of the heat-rejection or cooling systems described inconnection with FIGS. 1A, 1B, and 2.

The server rack itself may have a number of different implementations.Preferably, the mounting members are configured to mountably receive theplurality of servers in a vertical orientation, thereby minimizing thefootprint of the servers relative to the ground, and with the “front”¹panel facing upward for easy installation and removal of a serverwithout the need to remove or disturb any other server within the tank410. ¹ Upwards is defined as one of the two smallest sides of arectangular computer. The “back” is generally referred to as the sidewith wires inserted, such as power, communications, etc.

The mounting members may be also configured to mount the servers suchthat the top level 460 of the liquid coolant completely submerges thetop level 472 of the server rack 470 formed by the multiple servers 120.As a consequence, a volume of liquid coolant collects in a commonmanifold area above the server rack 470 to improve the circulation ofthe liquid coolant through the plurality of servers, thereby enhancingthe cooling of each respective server. The mounting members may also beconfigured to mount the servers in the server rack 470 above the bottomof the tank to create a volume of liquid coolant between each respectiveserver and the bottom of the tank such that the flow of the dielectricliquid coolant through the servers is improved. Preferably, the mountingmembers are configured to mount the servers closely adjacent to oneanother in the server rack to restrict the flow of the dielectric liquidcoolant between the vertically-oriented servers, such that the flow ofthe dielectric liquid coolant through the servers is enhanced.

A pump, such as pump 330 in FIG. 2, may pump liquid coolant from theexternal heat exchanger through the piping system into the tank 410 tomaintain coolant fluid circulation within the tank. The liquid coolantmay flow through each installed server and exit at the server sidepositioned opposite the inlet to the tank. In FIGS. 3 thru 6, the inletpiping 440 is located at one end of the rectangular tank 410 near thebottom of the tank; whereas the outlet piping 450 is located nearer thetop of the tank. This configuration permits the liquid coolant heated bythe heat generating components in the servers to naturally rise throughthe servers and exit through the top or “front panel” of the servers.

The servers may be configured to minimize mixing of the incoming liquidcoolant with outgoing liquid coolant. Each tank may be shaped (or have amember installed) to reduce the flow of coolant around the installedserver (e.g., to reduce by-pass flow), thereby improving coolant flowover each heat generating component and/or respective heat sink in eachof the multiple servers.

Alternatively, the location of the piping 440 and 450 may be reversedsuch that the heated liquid coolant may exit from the installed serversthrough its “rear” panel) into the outlet into the collection pipingsystem. The collection piping transports the heated liquid coolant tothe heat exchanger for rejecting at least some of the heat absorbed fromthe installed servers.

In another alternative rack design (not shown in the drawings), the tank410 is divided into a plurality of bins with each bin being sized toreceive one corresponding server with the “front panel” facing upward.The external pump pumps coolant from the external heat exchanger throughthe piping system into each bin to maintain a coolant fluid circulationwithin the tank and each respective bin. The liquid coolant may flowthrough each installed server and exit at a side positioned opposite theinlet to the tank and/or inlet to the bin. In addition, each bin may beconfigured to minimize mixing of the incoming liquid coolant withoutgoing liquid coolant. Each bin may be shaped (or have a memberinstalled) to reduce the flow of coolant around the installed server(e.g., to reduce by-pass flow), improving coolant flow over eachheat-generating component and/or respective heat sink in each of theservers.

FIGS. 7 and 8 depict another illustrative embodiment of a suitablefixture or server rack apparatus 500 for immersing a rack ofindependently operable servers in a liquid coolant 522 wherein thesurface of the liquid coolant 524 is above the top of the server rack.FIG. 7 shows an end elevation view of the apparatus 500, which includesa tub or tank 510 mounted on a mount 515 into which the servers 120 aresubmerged. The tank 510 may have an open top to form an open interiorvolume into which the servers may be mounted in a vertical orientationwith the front panel facing upward toward the open top of the tank. Thetank 510 is shaped and sized like the embodiment shown in FIGS. 3-6except as otherwise noted herein below. The inlet piping 540 is locatednear one end of one of the longer sides of the rectangular tank near thebottom of the tank. The output piping 550 is located at the opposite endof the opposing longer side of the rectangular tank also near the bottomof the tank. In this configuration, the fluid flow 560 of the liquidcoolant entering the tank through the inlet piping is initially throughthe volume 562 of liquid coolant formed by the longer side of the tankcontaining the inlet piping 540 and the side 572 of the server rack 570of servers 120 and then through the side 572 of the server rack throughthe servers 120 and out the opposite side 574 of the server rack into avolume 576 of liquid coolant formed by the side 574 of the server rackand the longer side of the tank containing the outlet piping 550.

FIG. 9 depicts an end elevation view of yet another illustrativeembodiment of a suitable fixture or server rack apparatus 600 for use inconnection with a combination of system 100 of FIG. 1A and system 200 ofFIG. 1B. In such a combination, there are two alternative modes ofoperating the cooling system for cooling the dielectric liquid coolantwherein the controller may switch the mode of operation depending on theenvironmental conditions. The tank 610 is shaped and sized like theembodiment shown in FIGS. 3-6 except as noted herein below. The tank 610also may have an open top to form an open interior volume into which theservers may be mounted in a vertical orientation with the front panelfacing upward toward the open top of the tank. The inlet piping 640 islocated nearer one end of one of the longer sides of the rectangulartank than the middle and is located nearer the bottom of the tank thanthe middle. The output piping 650 is located nearer the opposite end ofthe same longer side of the rectangular tank nearer the top of the tank.In the first mode of operation utilizing a mode of operation comparableto that of FIG. IA, the fluid flow 660 of the liquid coolant enteringthe tank through the inlet piping 640 is initially through the space 662formed by the bottom of the side of the tank containing the inlet piping640 and the bottom 672 of the server rack 670 of servers 120 and thenthrough the bottom 672 of the server rack through the servers 120 andout the front panel side 674 of the server rack into a space 676 formedby the top 674 of the server rack and the top surface 622 of the liquidcoolant and nearer the outlet piping 650. To permit the second mode ofoperation similar to FIG. 1B, a second heat exchanger 680 associatedwith an additional secondary cooling apparatus is mounted within thetank 610 and a second inlet piping 682 and a second output piping 684are inserted through the wall of the tank 610 and fluidly coupled to theheat exchanger to permit the flow of a separate second cooling fluidthrough the input piping 682, second heat exchanger 680, and outletpiping 684 back to the second secondary cooling apparatus.

In the second mode of operation, the pump associated with the first modeof operation is deactivated by the controller such that the fluidcircuit flow of the dielectric liquid coolant to the external heatexchanger of the first secondary cooling apparatus is deactivated. Nextthe internal heat exchanger 680 associated with the second alternativesecondary cooling apparatus is activated by the controller. In this modethe fluid flow of the dielectric fluid within the tank is reconfiguredsuch that the heated dielectric liquid coolant fluid flow 660 flowingout of the servers 120 does not flow out of the outlet piping 650.Instead, at least a portion of the liquid coolant fluid flow 660 isthrough the heat exchanger 680 to the bottom of the tank 610 and thenback through the servers 120. The heat rejected from the heat exchanger680 is thermally coupled to the second cooling fluid of the secondsecondary cooling system for dissipation or recovery.

FIG. 10 depicts an end elevation view of yet another illustrativeembodiment of a suitable fixture or server rack apparatus 700 for use inconnection with a combination of system 100 of FIG. 1A and system 200 ofFIG. 1B. In such a combination, there are two different modes ofoperating the cooling system for cooling the dielectric liquid coolant.The tank 710 is shaped and sized like the embodiment shown in FIGS. 3-6except as noted herein below. The tank 710 also may have an open top toform an open interior volume into which the servers 120 may be mountedin a horizontal orientation with the front panel facing toward theshorter side of the rectangular tank in which the inlet piping 740 islocated. The inlet piping 740 is located nearer one end of one of theshorter sides of the rectangular tank than the middle and is locatednearer the bottom of the tank than the middle. The output piping 750 islocated nearer the opposite end of the same shorter side of therectangular tank nearer the top of the tank. In the first mode ofoperation utilizing a mode of operation comparable to that of FIG. 1A,the fluid flow 760 of the liquid coolant entering the tank through theinlet piping 640 is initially through the space 762 formed by a longerside of the tank and the lower side 772 of the server rack 770 ofservers 120 and then through the bottom 772 of the server rack throughthe servers 120 and out the front panel 774 of the server rack into aspace 776 formed by the front 774 of the server rack and the shorterside of the tank nearer the outlet piping 750. To permit the second modeof operation similar to FIG. 1B, a second heat exchanger 780 associatedwith an additional secondary cooling apparatus is mounted within thetank 710 and a second inlet piping 782 and a second output piping 784are inserted through the wall of the tank 710 and fluidly coupled to theheat exchanger to permit the flow of a separate second cooling fluidthrough the input piping 782, second heat exchanger 780, and outletpiping 784 back to the second secondary cooling apparatus.

In the second mode of operation, the pump associated with the first modeof operation is deactivated by the controller such that the fluidcircuit flow of the dielectric liquid coolant to the external heatexchanger of the first secondary cooling apparatus is deactivated. Nextthe internal heat exchanger 780 associated with the second alternativesecondary cooling apparatus is activated by the controller. In this modethe fluid flow of the dielectric fluid within the tank is reconfiguredsuch that the heated dielectric liquid coolant fluid flow 760 flowingout of the servers 120 does not flow out of the outlet piping 750.Instead, at least a portion of the liquid coolant fluid flow 760 isthrough the heat exchanger 780 to the bottom of the tank 710 and thenback through the servers 120. The heat rejected from the heat exchanger780 is then thermally coupled to the second cooling fluid of the secondsecondary cooling system for dissipation or recovery.

A combination of the system 100 and 200 using the alternative serverrack apparatus of FIG. 9 and FIG. 10 that permit two different modes ofoperating the server rack cooling system for cooling the dielectricliquid coolant may be useful in certain applications and climates, forexample, in an arid climate having cool nights and very hot days. Duringthe cool days, the combination system employing the embodiments of FIG.9 or FIG. 10 may be used in a first mode similar to that of FIG. 1Awherein the dielectric fluid is fluidly coupled to an external heatexchanger associated with a radiator-type secondary cooling system.During the hot days, the combination system may be used in a second modesimilar to that of FIG. 1B wherein the dielectric liquid coolant isfluidly coupled through the internal heat exchanger, which is associatedwith a second secondary cooling apparatus, such as a vapor-compressioncycle refrigeration cooling system.

FIGS. 11, 13 and 14 depict another illustrative embodiment of a suitablefixture or rack apparatus 800 for immersing side-by-sideimmersion-cooled server racks of standard commercially availableversions of independently operable servers, such as those depicted inFIG. 12A for example, in a liquid coolant 822 with the electricalconnections to the servers shown. The tank 810 may face upward with anopen top 812 to form an open interior volume and may be shaped to have alength L, width W, and height H with the minimum footprint to insert tworows or racks 830 and 832 of multiple servers 820. The tank 810 may beshaped and the dimensions sized such that multiple standard-sizedservers 820, typically measured in units of “U” or 1.75 inches (as shownin FIG. 12A), can be supported in two racks without significantmodification. Suitable mounting members may be used to mount the serversin the tank to configure the server rack 830 and 832 within the tank.Specifically, the mounting members (not shown) may be fixedly attachedalong the length L of each longer side of the tank 810 and in the middleof the tank 810 between the two shorter ends of the tank to support therack ears 836 of a standard rack-mountable server 820 shown in FIG. 12A.

The tank may be fabricated to have an inlet pipe or line from a pipingsystem connected to a heat exchanger for the flow of lower temperatureor cooled liquid coolant into the tank 810 and an outlet pipe or lineconnected to collection piping for the flowing or pumping of heatedcoolant out of the tank to the distally located heat exchanger as shownin FIG. 3. After the two racks of multiple servers are mounted insidethe tank 810, the level 824 of the liquid coolant 822 may be carefullycontrolled to adjust the amount of flow of the liquid coolant throughthe multiple servers and to adjust the amount of heat removal from theheat generating electronic components in the servers.

Orienting the servers in a vertical orientation with the front panelfacing upward may also be advantageous due to the typical movable harddrive installation in a standard commercially available server. When astandard server such as shown in FIG. 12A, is oriented vertically thehard drive 890 of such a server, as shown in FIG. 12B, is orientedvertically with the cables connecting at the bottom of the drive. Insome embodiments, a liquid-resistant or liquid-proof enclosure 892 forthe movable hard-drive 890 in each of the servers 820 can be insertedover the hard-drives prior to the submersion of the server into thedielectric liquid coolant to protect moving components (e.g., a platen)from being damaged by the viscous liquid coolant. The previouslyinserted liquid-proof enclosure traps air within the hard drive. Theentrapped air prevents the dielectric liquid coolant from entering theportion of the disk drive containing the movable disk.

As shown in FIG. 13, the apparatus 800 also may have cable trays 840mounted along two sides of the tank 810 paralleling the sides of theserver racks 830 and 832 to organize the signal and control networkcabling 842 from the servers to the controller and other computers inthe data center and beyond. The apparatus 800 may further have powerdistribution units (“PDUs”) 844 mounted above the space between theserver racks in order to distribute needed electrical power throughsuitable power cables 846 to the multiple servers.

The server racks 830 and 832 may have a number of differentimplementations, some of which affect the flow characteristics of theliquid coolant. Preferably, the mounting members are configured tomountably receive the plurality of servers in a vertical orientation,thereby minimizing the footprint of the servers relative to the ground,and with the “front”² panel facing upward for easy installation andremoval of a server without the need to remove or disturb any otherserver within the tank 810. ² Upwards is defined as one of the twosmallest sides of a rectangular server. The “back” is generally referredto as the side with wires inserted, such as power, communications, etc.

As shown in FIGS. 12 and 14, the mounting members may be also configuredto mount the servers such that the top level 824 of the liquid coolant822 completely submerges the top level 872 of the server rack 830 and832 formed by the multiple servers 820. As a consequence, a volume ofliquid coolant collects in a common manifold area above each of theservers to improve the circulation of the liquid coolant through theplurality of servers, thereby enhancing the cooling of each respectiveserver. The mounting members may also be configured to mount the serversin the server rack 830 and 832 above the bottom of the tank 810 tocreate a volume of liquid coolant between each respective server 820 andthe bottom of the tank such that the flow of the dielectric liquidcoolant through the plurality of servers is improved. Preferably, themounting members are configured to mount the servers closely adjacent toone another in the server rack to restrict the flow of the dielectricliquid coolant between the plurality of vertically-oriented servers,such that the flow of the dielectric liquid coolant through theplurality of servers is enhanced.

The tank may also be sized and shaped to minimize the mixing of the cooland heated liquid coolant. Further the apparatus 800 may include aremovable top so that in the event of fire the top of the fixtureapparatus may be enclosed to smother the fire.

A pump, such as the pump 330 of FIG. 2, may pump liquid coolant from theexternal heat exchanger through the piping system into the tank 810 tomaintain the coolant fluid flow within the tank. The liquid coolant mayflow through each installed server and exit through the outlet pipe fromthe tank. Similar to FIGS. 3 thru 6, the inlet piping may be located atone end of the rectangular tank 810 near the bottom of the tank; whereasthe outlet piping may be located nearer the top of the tank. Thisconfiguration permits the liquid coolant heated by the heat generatingcomponents in the servers to naturally rise through the servers and exitthrough the front panel of the servers. Because the flow is relativelylow in comparison to the total volume of the container, the fluidconducts to be relatively uniform temperature.

Alternatively, the location of the inlet and outlet piping may bereversed such that the heated liquid coolant may exit from the installedservers through its “rear” panel) into the outlet into the collectionpiping system. The collection piping transports the heated liquidcoolant to the heat exchanger for rejecting at least some of the heatabsorbed from the installed servers.

In commercially available servers, fans are often installed within theservers for distributing a cooling medium (e.g., air) among componentsand regions within the server. In some embodiments, these fans can helpdistribute a liquid coolant among the components and regions within theservers. Coolant flow rate and/or fan-speed can be adjusted in responseto a component temperature excursion above a pre-determined threshold,or even a computational workload, to maintain component temperatures ator below a maximum specified (as by, for example, the componentmanufacturer) temperature, while at the same time maintaining a coolanttemperature at an elevated temperature, such as at the highest coolanttemperature that still maintains component temperatures below a maximumthreshold. Fan speed can be modulated, but does not have to be.

Additional fluid velocity augmentation devices, such as multiple fans880 may be mounted under each of the server racks 830 and 832 in thevolume of liquid coolant between the plurality of servers in eachrespective rack and the bottom of the tank to increase the mixing of thedielectric liquid coolant within the tank, and improving the flow of thecoolant through the plurality of servers. Other suitable fluidaugmentation devices include nozzles mounted on the end of a line fromthe cooling inlet piping which may be directed toward the desired entrypoint of the liquid coolant into the servers to enhance the fluidvelocity of the liquid coolant through the servers.

FIG. 14 shows the fluid flow 860 of the liquid coolant 822 through theservers 820 in the apparatus 800 in more detail. For the serverconfiguration shown, the fluid flow 860 of the liquid coolant enteringthe tank through the lower inlet piping is initially directed through avolume of liquid coolant 862 formed by the bottom of the side of thetank containing the inlet piping and the bottom 872 of the server rack830 and 832 of servers 820 and then through the bottom 872 of the serverrack through the servers 820 and out the top side 874 of the server rackinto a volume of liquid coolant 876 formed by the top 874 of the serverrack and the top surface 822 of the liquid coolant and near the outletpiping located near the top of the tank.

In summary, the immersion of servers into a liquid coolant within thefixture apparatus various embodiments 400, 500, 600, 700, and 800 of thefixture apparatus shown in FIGS. 3-14 reduces the temperature differencebetween server electronic components generating heat and the liquidcoolant medium used to cool them. Preferably, the median coolanttemperature can be kept at as high a level as possible while maintaininga component temperature during operation below its specified maximumallowable operating temperature. Such a high-temperature cooling mediumprovides sufficient cooling while reducing the power consumed to coolthe electronic components, as compared to cooling the component with alower-temperature cooling medium such as refrigerated air.

Therefore the fixture apparatus for submerging the servers in adielectric liquid coolant provides for the following advantages:

-   -   designed to maximize fluid temperature through flow control    -   permits the use of standard commercially available rack        mountable servers originally designed for air cooling with        minimal modification from commercially available configurations    -   transfers heat from all heat-generating components into the        dielectric liquid coolant without the addition of cold plates,        piping or additional parts internal to the servers    -   has an open top which enables the removal of any server without        the removal of a different server (e.g., servers remain        independently operable)    -   only requires the tank enclosure to be sealed rather than        needing to hermetically seal each of the individual servers        being mounted in the server racks    -   guides the fluid flow such that cool liquid coolant flows in and        heated liquid coolant flows out of the servers    -   may use fluid velocity augmentation, such as fan speed        modulation, to enhance the flow of the liquid coolant through        each server    -   improves the installed density of servers in a conventional        server room or data center by minimizing the footprint of the        servers relative to the ground    -   uses a controller (i) to monitor temperature and flow conditions        in the fixture apparatus and the power consumption of the        servers and cooling system to minimize the amount of power        required to cool the servers and (ii) to control the heat        exchange method, thereby enabling the data center to recapture        heat, if desirable, or dissipate the heat in the most efficient        manner when heat recapture is not desirable.

FIG. 15 depicts a schematic illustration of a system for coolingmultiple immersion-cooled server racks of the type shown in, forexample, FIG. 3, located in a server room of a typical data center. Thecooling system includes multiple server racks 310 fluidly coupled inparallel through respective outlet piping 315 to collection pipingsystem 902. Collection piping 902 collects the heated liquid coolantflowing out of the multiple server racks. The collection piping 902, inturn, is fluidly coupled to a pump 904 which pumps the collected heatedliquid coolant through piping 906 to a fluid line 908 in a heatexchanger 910. The heated liquid coolant in fluid line 908 is thermallycoupled to a cooling fluid flowing in line 912 through heat exchanger910. The cooling fluid in line 912, in turn, is coupled to a selectedone of the heat rejection or cooling apparatus 352, 354, 356, etc aspreviously described for either dissipating or recovering the heatabsorbed by the cooling fluid from the heated liquid coolant.

The cooled liquid coolant exiting from line 908 of the heat exchanger910 is then fluidly coupled through distribution piping system 914 to aplurality of parallel piping 916 fluidly connected to valves 918. Valves918, in turn, are fluidly connected in parallel to the inlet piping 370to the multiple server racks 310.

The controller 920 may receive monitoring signals of the temperature ofthe heated liquid coolant exiting the server racks through control lines924. The controller may also receive monitoring signals of the flow rateof the liquid coolant at various locations in the piping 902 throughcontrol lines 925 and the flow rate through the pump 904 through controllines 926. The controller 920 may also receive monitoring signalsrelating to the type of secondary cooling apparatus selected and theflow rate of the cooling fluid in the selected secondary coolingapparatus through control lines 928.

As previously described, the controller 920 operates an applicationprogram that processes the information received from the variousmonitoring signals to selected an optimum elevated temperature, theenergy needed to be rejected by the system to cool the servers andmaintain the elevated temperature, and then determine the varioussettings of the system 900 components that will be needed to maintainthe elevated temperature of the liquid coolant exiting the servers inthe multiple server racks 310. The various components of the system 900controlled by the controller 920 include any fluid velocity augmentationdevices positioned below the server racks, the pump 904, valves 918, thevalve 390 (FIG. 2) for switching the flow of the heated liquid coolantbetween secondary cooling apparatus to be used, and the selectedsecondary cooling apparatus.

The controller may adjust the flow of the cooled liquid coolant througheach of the valves 918 to adjust the volume of the flow of the cooledliquid coolant among the different server racks 310.

The controller 920 may control any fluid velocity augmentation devicesin the server racks through control lines and may also control thepumping rate of the pump 904 through control line 930. In addition, thecontroller 920 through control line 932 may select one of a plurality ofsecondary cooling apparatus 352, 354, 356, etc to optimize the secondarycooling apparatus to the environmental and server rack conditions andcontrol the amount of heat being rejected by the selected secondarycooling apparatus by adjusting the flow of the cooling fluid in thesecondary cooling apparatus.

Methods of Operation

FIG. 16 illustrates an exemplary method of cooling one or moreindependently operable servers at least partially immersed within aliquid coolant inside a tank with an open interior volume. This methodmay be used to implement the systems of FIG. 1A or 1B. The methodincludes a step 10 of flowing a dielectric liquid coolant in a fluidcircuit through the plurality of servers immersed within the dielectricliquid coolant for absorbing at least a portion of any heat beingdissipated by the servers. In step 12, the temperature of the liquidcoolant at at least one location is monitored by a controller. In step14, the controller determines what temperature would be the optimumelevated temperature of the heated dielectric liquid coolant as it exitsthe plurality of servers such that the exiting liquid coolantsufficiently cools the plurality of servers while reducing the amount ofenergy consumed to sufficiently cool each respective server. Aspreviously described, the determined optimum elevated temperaturepreferably is a temperature significantly higher than the typicalcomfortable room temperature for humans and lower than the maximumpermissible temperature of the most sensitive heat generating electroniccomponent in the servers. In step 16, the controller periodicallydetermines the energy needed to reject the heat absorbed by the liquidcoolant and maintain the liquid coolant exiting the servers at theelevated temperature. In step 18, the optimum secondary coolingapparatus to minimize the amount of energy needed to be consumed tomaintain the elevated temperature and cool the servers is selected. Instep 20, the liquid coolant heated by the servers is thermally coupledto a heat exchanger. In step 22, a portion of the heat absorbed by theliquid coolant from the servers is rejected through the heat exchanger.In step 24, in response to the energy consumption periodicallydetermined, the amount of heat rejected through the heat exchanger isperiodically adjusted such that the liquid coolant exiting the pluralityof servers at the elevated temperature sufficiently cools the pluralityof servers while reducing the amount of energy consumed to sufficientlycool each respective server.

It should be noted that it may be desirable to also monitor (i) thetemperature of the liquid coolant at multiple locations, (ii) the flowrate of the liquid coolant through the fluid circuit; (iii) thetemperature of the electronic components of the respective servers byconnecting the temperature signals outputted by standard commerciallyavailable servers to the controller; and the power consumption of theservers through signals outputted from the servers to the controller.

In response to the energy consumption periodically determined and theflow rate, the controller may periodically adjust the pumping rate ofthe liquid coolant through the pump and the heat exchanger such that theliquid coolant exiting the servers at the elevated temperaturesufficiently cools the plurality of servers while reducing the amount ofenergy consumed to sufficiently cool each respective server.

In connection with the operation of the cooling system depicted in FIG.1A and further depicted in FIGS. 2, the heat exchanger for directlyrejecting heat from the liquid coolant is located externally to thefixture apparatus and the method employs a first type of thermodynamiccycle. In this embodiment, the step of thermally coupling the liquidcoolant to a heat exchanger includes the step of fluidly coupling theliquid coolant to a distally located heat exchanger and the flow of theliquid coolant passes through outlet piping in the tank into a fluidcircuit that is partially outside the tank. A more detailed descriptionof the steps occurring in this embodiment is set forth below inconnection with the description of FIGS. 17A and 17B.

In connection with the operation of the cooling system 200 in FIG. 1B,the coupler, such a heat exchanger, for directly rejecting heat from theheated liquid coolant flowing through the servers 120 is locatedinternally to the tank 210. The method of operation of this system 200employs a second type of thermodynamic cycle. In this alternative systemembodiment, the method include the steps of flowing at least a portionof the cooler liquid coolant in a first fluid portion of a first liquidcircuit through each of the plurality of servers wherein the liquidcoolant exiting the plurality of servers is heated to an elevatedtemperature; thermally coupling the heated liquid coolant through acoupler to a cooling fluid located in a first portion of a second fluidcircuit; fluidly coupling the heated cooling liquid in the first portionof the second fluid circuit to an external distally located heatexchanger for rejecting at least a portion of the heat coupled throughthe second liquid circuit from the heated dielectric liquid coolant;fluidly coupling the cooled cooling fluid from the distally located heatexchanger through a second portion of the second liquid circuit to thecoupler; thermally coupling the cooled cooling fluid through the couplerto the first portion of the first liquid circuit.

This method may also include the steps of monitoring the flow rate ofthe cooling fluid in the second fluid circuit; and monitoring thetemperature of at least one of the heat-generating electronic componentsin each respective server; periodically determining the energy needed tocool the servers by the cooling of the heated cooling fluid to thecooler temperature. This method may also include the step of enhancingthe fluid velocity of the dielectric fluid through the servers usingfluid velocity augmentation devices, such fans or nozzles, as previouslydescribed herein.

In response to the controller periodically determining the energy neededto reject the absorbed heat and the flow rate of the cooling liquid, themethod may also include the step of periodically adjusting the flow rateof the cooling liquid through the second fluid circuit such that theliquid coolant exiting the servers at the elevated temperaturesufficiently cools the servers while reducing the amount of energyconsumed to sufficiently cool each respective server. The method mayfurther include the steps of monitoring the temperature of the coolingfluid in the second fluid circuit.

It should be noted that in the system employing the second type ofthermodynamic cycle, the flow of the liquid coolant is contained insidethe tank in which the servers are submerged. Preferably the fluid flowin this first fluid circuit is from the bottom of the server through theserver to the top thereof, where heated liquid coolant exists. Once thecoolant exits the top of the server, the coolant is cooled by passing itthrough the heat exchanger in the liquid coolant. Once cooled, theliquid coolant sinks to the bottom of the tank. The flow of the coolantin the first fluid circuit can be supplemented by fans, internal orexternal to the servers. In the preferred embodiment, cooling takesplace near the exiting of the heated coolant from the servers.

FIG. 17A illustrates the physical steps in the method of cooling one ormore independently operable servers immersed in tank of liquid coolantemploying the system of FIG. 1A or FIG. 3. In step 24 of the method,liquid coolant flows into the tank with the servers. In step 26, thedielectric liquid coolant flows in a fluid circuit through the pluralityof servers immersed within the dielectric liquid coolant for absorbingat least a portion of any heat being dissipated by the servers. In step28, the fluid velocity of the liquid coolant may be optionally enhancedby using fluid velocity augmentation devices, such as fans, in andoutside of the servers. In step 30, the temperature of the liquidcoolant is monitored at least one location within the fluid circuit. Instep 32, a secondary cooling system is selected to minimize energyusage. In step 34, the liquid coolant heated by the servers pumped to aheat exchanger distally located from the tank. In step 36, at least aportion of the heat absorbed by the liquid coolant is rejected throughthe heat exchanger. In step 38, the cooled liquid coolant is fluidlycoupled back to the tank. In step 40, the fluid flow in the secondarycooling apparatus is adjusted to aid in maintaining the elevatedtemperature. In step 42, the rejected heat is dissipated through theselected secondary cooling apparatus or in step 44, the rejected heat isrecovered by the selected secondary cooling apparatus.

FIG. 17B illustrates the computer controller-based steps in the methodof cooling one or more independently operable servers immersed in tankof liquid coolant employing the system of FIG. 1A or FIG. 3. In step 52,the controller receives signals relating to the system operation fromvarious sensors relating to temperature, fluid flow, and powerconsumption. In step 54, the controller determines the optimum elevatedtemperature for cooling the servers. In step 56, the controllerperiodically determines the energy needed to cool the plurality ofservers. In response to the energy consumption periodically determined,the controller in step 58 periodically determining the optimal secondarycooling method to minimize energy usage in order to adjust the amount ofheat to be rejected through the heat exchanger such that the liquidcoolant exiting the plurality of servers at the elevated temperaturesufficiently cools the plurality of servers while reducing the amount ofenergy consumed to sufficiently cool each respective server. In step 60,the controller determines the preferable settings for the dielectricliquid coolant pump, type of secondary cooling apparatus, and optionallythe fluid velocity of the liquid coolant in the tank. In step 62, thecontroller executes the output control signals to the pumps, valves, andfluid velocity augmentation systems, i.e. fans or nozzles. In step 64,the controller provides a failure notification in the event the systemfails to operate as planned. For example the controller provides afailure notification is there is a safety issue or the system is downfor any reason.

In summary, the implementation of the methods disclosed in the exemplaryalternate embodiments described herein for cooling server racks immersedin a dielectric liquid coolant by maintaining an elevated temperaturecan minimize the amount of power required to cool the servers. Thisaccomplished by taking advantage of the number of irreversibility's ortemperature differences present in a normal server cooling system thatcan be reduced to improve cooling efficiency. The reduction oftemperature differences between the incoming cool liquid coolant and theheated outgoing liquid coolant is made possible by:

controlling the amount of liquid coolant flow to each server by usingspeed modulated fluid velocity augmentation devices to ensure flow issufficient to cool components with changing demand; and

a controller maintaining coolant temperature at the maximum allowabletemperature (e.g., between 90 and 130 degrees F.) by using the efficientheat removal methods described. The computer controller doesn'tnecessarily have to separate from the servers that are being cooled.

The reduction of irreversibility's in the thermodynamic cycle increasesefficiency and therefore reduces overall power consumed. With thedescribed features, it should be possible to safely maintain fluidtemperatures at approximately 105 F, significantly higher than roomtemperature and the maximum US average outdoor temperature by month (75degrees F. during summer). At this temperature, heat can be dissipatedwith minimum power or recaptured by heating other unrelated componentssuch as building hot water or ambient indoor air in cold climates.Further, this method should minimize or remove the need forenergy-intensive thermal processes associated with the current methodsof server/computer cooling, which include refrigeration as the primarymode of heat dissipation. If heat dissipation (versus heat recapture) isdesired, an elevated coolant temperature allows methods requiring up to⅛ or less power than conventional refrigeration methods. These lowenergy methods can include direct fluid to air heat exchangers,evaporative cooling, or other similar methods. Refrigeration, however,can be used to supplement cooling methods disclosed herein whileconsuming a minimum power.

Although the invention has been described with reference to specificembodiments, these descriptions are not meant to be construed in alimiting sense. Various modifications of the disclosed embodiments, aswell as alternative embodiments of the invention will become apparent topersons skilled in the art upon reference to the description of theinvention. It should be appreciated by those skilled in the art that theconception and the specific embodiment disclosed may be readily utilizedas a basis for modifying or designing other structures for carrying outthe same purposes of the present invention. It should also be realizedby those skilled in the art that such equivalent constructions do notdepart from the spirit and scope of the invention as set forth in theappended claims.

It is therefore, contemplated that the claims will cover any suchmodifications or embodiments that fall within the true scope of theinvention.

1-52. (canceled)
 53. An apparatus for holding and cooling rack-mountableservers containing heat-generating electronic components, comprising: atleast one tank defining an open interior volume and having a coolantinlet for receiving a dielectric liquid coolant within the open interiorvolume and having a coolant outlet for allowing the coolant to flow fromthe open interior volume, the coolant inlet and the coolant outlet beingfluidly coupled to each other; and one or more mounting memberspositioned within the interior volume and configured to hold a pluralityof rack-mountable servers in a vertical orientation within the interiorvolume such that the plurality of rack-mountable servers can be commonlysubmersed in a volume of the dielectric liquid coolant in the at leastone tank and such that, when the plurality of rack-mountable servers arecommonly submersed in a volume of the dielectric liquid coolant, atleast a portion of the dielectric liquid coolant can be vertically movedacross heat producing components in at least two of the rack-mountableservers, wherein the rack-mountable servers are mountable in the atleast one tank such that at least one of the rack-mountable servers isindependently removable from a volume of dielectric liquid coolant andfrom the at least one tank without the need to remove the otherrack-mountable servers from the volume of dielectric liquid coolant inthe at least one tank and such that the other rack-mountable servers canremain operating while submersed in the volume of dielectric liquidcoolant.
 54. The apparatus of claim 53, wherein the tank comprises anopen top, wherein the rack-mountable servers are mountable such that atleast one of the rack-mountable servers can be removed from the tankthrough the open top while the other rack-mountable servers remain atleast partially submersed in the dielectric liquid coolant and inoperation.
 55. The apparatus of claim 53, further comprising acontroller configured to maintain liquid coolant exiting therack-mountable servers at a temperature that is significantly higherthan comfortable room temperature and lower than the maximum permissibletemperature of the most sensitive heat generating electronic componentof the rack-mountable servers.
 56. The apparatus of claim 53, wherein atleast one of the rack-mountable servers submersed in the dielectricliquid coolant comprises an unsealed case housing two or more electroniccomponents of the at least one rack-mountable server.
 57. The apparatusof claim 53, wherein at least one of the mounting members is configuredto receive standard rack-mountable servers from at least two differentsources such that the commodity servers are commonly in the volume ofdielectric liquid coolant.
 58. The apparatus of claim 53, furthercomprising one or more pumps configured to move a portion of thedielectric liquid coolant across the other rack-mountable servers whilethe at least one rack-mountable server is removed from the volume ofdielectric liquid coolant.
 59. The apparatus of claim 53, wherein therack-mountable servers are mountable to allow vertical flow across heatproducing components of the rack-mountable servers.
 60. The apparatus ofclaim 53, further comprising a pump configured to move dielectric liquidcoolant in the at least one tank, wherein the rack-mountable servers arearranged to allow the pump to produce parallel vertical flow of thedielectric liquid coolant across heat producing components in at leasttwo of the rack-mountable servers commonly submerged in the volume ofdielectric liquid coolant.
 61. The apparatus of claim 53, furthercomprising a pump configured to move dielectric liquid coolant in the atleast one tank, wherein the rack-mountable servers are arranged to allowthe pump to produce parallel bottom-to-top flow of the dielectric liquidcoolant across heat producing components in at least two of therack-mountable servers commonly submerged in the volume of dielectricliquid coolant.
 62. The apparatus of claim 53, wherein the dielectriccoolant comprises mineral oil.
 63. The apparatus of claim 53, wherein atleast one of the rack-mountable servers comprises an enclosure, whereinflow of the dielectric liquid coolant around the plurality ofrack-mountable servers is restricted such that flow through theenclosure of the at least one rack-mountable server is enhanced.
 64. Theapparatus of claim 53, wherein the apparatus comprises two or moreracks, wherein each of the racks comprises a tank configured to hold aplurality of rack-mountable servers, further comprising one or morepumps, wherein a first one of the pumps is configured to move dielectricliquid coolant across heat producing components in at least two of therack-mountable servers.
 65. The apparatus of claim 53, wherein, when theplurality of rack-mountable servers are mountably received and thedielectric liquid coolant is moved in the at least one tank, at leasttwo of the respective rack-mountable server is completely submergedwithin the dielectric liquid coolant such that a volume of dielectricliquid coolant collects in a common area above the plurality ofrack-mountable servers and at or near the surface of the dielectricliquid coolant.
 66. The apparatus of claim 53, further comprising a pumpconfigured to move dielectric liquid coolant in the at least one tank,wherein at least some of the rack-mountable servers are held in a row,wherein, when the plurality of rack-mountable servers are mountablyreceived in the at least one tank and the dielectric liquid coolant ismoved within the at least one tank, heated dielectric liquid coolant atthe surface of the volume of dielectric liquid coolant is movedcrossways relative to the direction of the row of rack-mountableservers.
 67. The apparatus of claim 53, wherein: the one or moremounting members are configured to mountably receive the plurality ofrack-mountable servers above the bottom of the at least one tank to forma volume between each respective rack-mountable server and the at leastone tank to permit the flow of dielectric liquid coolant through theplurality of rack-mountable servers.
 68. The apparatus of claim 53,further comprising: a secondary cooling circuit comprising a secondliquid coolant, and a liquid-to-liquid heat exchanger, wherein theliquid-to-liquid heat exchanger is configured transfer heat from thedielectric liquid coolant to the second liquid coolant, wherein thesecondary cooling circuit is configured to reject heat to a locationdistal to the at least one tank.
 69. The apparatus of claim 53, furthercomprising a plurality of nozzles configured to direct dielectric liquidcoolant from the cooling inlet piping toward a desired entry point intothe rack-mounted servers to augment the fluid velocity of the liquidcoolant through the rack-mounted servers.
 70. A method for maintainingrack-mountable servers containing heat-generating electronic components,comprising: removing one or more rack-mounted servers from a volume ofdielectric liquid coolant in a rack while one or more other rack-mountedservers remain operating and submersed in the volume of dielectricliquid and while a portion of the dielectric liquid coolant is movedvertically over heat producing components of at least one of therack-mountable servers; and replacing or reinstalling, while the otherrack-mounted servers remain submersed in the volume of dielectric liquidcoolant and in operation, at least one of the removed rack-mountableservers.
 71. The method of claim 70, further comprising moving, whilethe one or more rack-mounted servers are being removed, at least aportion of the dielectric liquid coolant across heat producingcomponents in at least one of rack-mountable servers remaining submersedin the rack.
 72. The method of claim 70, further comprisingdisconnecting the replaced server and connecting the replacement server.